Papers with multimodal counterparts
M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models and their multimodal counterparts have shown significant performance disparities across different languages and cultural contexts. |
| Approach: | They propose to evaluate LLMs on diverse vision-language tasks within a multilingual and multicultural context using M5 benchmark. |
| Outcome: | The proposed benchmarks highlight task-agnostic performance disparities between languages and cultural contexts. |
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Vatsal Sharan, Robin Jia, Willie Neiswanger
| Challenge: | Steering methods have emerged as effective tools for guiding large language models’ behavior, yet multimodal large language model lacks comparable techniques due to architectural diversity and limited availability of multimodal steering vectors. |
| Approach: | They validate steering vectors derived solely from text-only LLM backbones and use a cross-modal transfer technique to reuse existing interpretability tools. |
| Outcome: | The proposed steering vectors can guide and enhance multimodal models using SPAR, Mean Shift, and Linear Probing. |